Local vs. global memory in the IBM RP3: experiments and performance modelling

A number of experiments regarding the placement of instructions, private data and shared data in the Non-Uniform Memory Access multiprocessor, RP3 has been performed. Three Scientific/Mathematical workloads have been used in the experiments, and the results have been modelled in a simple performance model which takes linear contention into consideration. The results indicate that it can very well be feasible not to have memory local to the processors in RP3-like archi-tectures. There seems to be a trade-off between the effort spent in the design on the memory system and the intercon-nection network and the use of local memory which can be costly in terms of prohibited process migration and more complicated software management. In the construction of highly parallel multiprocessors, different approaches have been adopted in order to achieve efficient access to main memory. Some use Uniform Memory Access (UMA) making sure that performance is not degraded by having processor cache memories. Others which have Non-Uniform Memory Access (NUMA) achieve this by placing pieces of the memory space in memory close to a processor, thus reducing the traffic in the interconnection network. The different architectures require different techniques to reduce network and memory contention. The IBM RP3 multiprocessor can be said to belong to both of these classes with its capability to configure its memory as local memory, costly to access by remote processors , and/or global memory which is almost uniformly accessible by all processors. It has processor caches and a high bandwidth, low-latency multistage interconnection network. One of the most commonly mentioned advantages of having shared memory multiprocessors is for their ease of programming, and the question of load balancing. Ease of programming because of the insignificance of where shared data are located, and with shared memory equally accessible from all processors, it makes no difference in which processor a specific thread (The term thread is in this paper used denoting the smallest concurrently executable unit, see Section 2.2) executes. A thread may even move around between processors in order to utilize them more efficiently. A uniform memory access multiprocessor architecture has the processors on one side of the interconnection network , and the memory on the other side. All memory references has to go through the interconnection network. There may be local processor caches in order to minimize the traffic through the interconnection network. In contrast, the NUMA architecture there is memory close to each processor accessible …